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Section  I.  Test  and  Evaluation  ( DT&E,  IOT&E,  OT&E,  FOT&E ) 

1-1.  Pre-Testing 

a.  Review  AFR  80-14  for  an  understanding  of  responsibilities. 

b.  Review  Test  and  Evaluation  Objective  Annex  (TEOA)  of  the  PMD. 

c.  Review  T&E  section  of  the  PMP  and  TEMP. 

d.  Review  the  Computer  Program  Configuration  Item  Development  Test  and 
Evaluation  Test  Report  Checklist,  Figure  IV-1-1. 

1-2.  Testing  Conduct 

a.  Participate  in  T&E  test  conduct  functions  as  a  member  of  the  P0  Test 

Team. 

b.  Coordinate  with  using/supporting  (AFTEC)  commaod  participants  and 
resolve  comments  as  appropriate.  AFTEC  responsibilities  are  contained  in  AFR 
23-36. 

1-3-  Post-Te3tiog  Activities 

c.  Review  and  approve  the  test  reports  pertinent  to  T&E. 

d.  Monitor  resolutions  of  program  trouble  reports  opened  during  any  T&E 

test. 
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Figure  IV-1-1  COMPUTER  PROGRAM  CONFIGURATION  ITEM  (CPCI)  DEVELOPMENT 
TEST  AND  EVALUATION  TEST  REPORT  CHECKLIST 

This  checklist  for  reviewing  CPCI  Development  Test  and  Evaluation  (DT&E) 
Test  Reports  is  based  on  the  requirements  stated  in  ESD/ALEQ's  June  1978  Data 
Item  Description  (DID)  Computer  Program  Configuration  Item  (CPCI)  Development 
Test  and  Evaluation  Test  Report  DID(U)-E-743.  In  reviewing  a  CPCI  DT&E  test 
report,  the  reviewer  must  also  be  aware  of  any  supplemental  or  contrary 
direction  for  preparation  of  the  computer  program  test  report  contained  in  the 
Statement  of  Work  (SOW),  in  the  Contract  Data  Requirements  List  (CDRL)  or  in 
CDRL  backup  sheets. 

The  cnecklist  questions  are  numbered  sequentially  according  to  the 
preparation  instructions  contained  in  the  cited  Data  Item  Description.  The 
parenthetical  numbers  following  the  questions  refer  to  the  related  paragraphs 
of  the  instructions. 
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1.  CPC I  DT&E  TEST  REPORT  REQUIREMENTS 

Yes  No 

1)  Has  a  separate  test  report  beeo  submitted  for  preliminary  (  )  (  ) 

Qualification  Testiog  (PQT)  and  Formal  Qualification 

Testing  ( FQT ) ?  (l.a) 

2)  Is  an  incremental  draft  of  each  test  report  being  sub-  (  )  (  ) 

mitted  in  accordance  with  the  planned  test  phases  of 

increments?  (l.b) 

3)  Was  each  report  submitted  following  the  last  test  in  the  (  )  (  ) 

phase  within  the  number  of  days  specified  on  DD  Form  1423? 

(l.b) 

4)  Does  the  final  version  of  the  PQT  Test  Report  consist  of  (  )  ; 

a  summation  report  of  the  total  PQT  activities?  (l.c) 


5)  Does  the  final  version  of  the  FQT  Test  Report  coosist  of 
a  summation  report  of  the  total  FQT  activities?  (l.c) 

6)  Have  all  incremental  reports  previously  submitted  and 
revised,  in  final  form,  beeo  resubmitted?  (l.c) 


2.  COMPUTER  PROGRAM  CONFIGURATION  ITEM  IDENTIFICATION 

7)  Has  the  CPCI  to  which  the  test  applies  been  identified 
by  number?  (2. a) 

8)  Has  the  CPCI  to  which  the  test  applies  beeo  identified 
by  approved  nomenclature?  (2. a) 


2.b.  TEST  IDENTIFICATION 

9)  Is  the  identification  of  the  individual  qualification 
test  as  showo  on  the  test  procedure  shown  on  the  test 
report?  (2.b) 


2.C.  PRIMARY  FUNCTION 

10)  Have  all  the  CPCI' 3  primary  functions  or  Computer  Program 
Components  to  which  the  test  applies  beeo  identified? 
(2.c) 

2.d.  TEST  PLAN  AND  TEST  PROCEDURES  REFERENCE 

11)  Has  the  DT4E  CPCI  Test  Plan  for  the  test  been  referenced. 
(2.d) 


(  )  i  ) 

(  )  (  ) 

(  )  (  ) 

(  )  (  ) 

(  )  (  ) 


12)  Have  the  DT4E  CPCI  Test  Procedures  for  the  test  beeo 
referenced?  (2.d) 


(  )  (  ) 
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Ye  3  No 

2.e.  TEST  RESULTS 

13)  Witn  tne  exception  of  appropriate  identification  of  (  )  (  ) 

differences  in  aod  exclusions  of  planned  test  objectives, 

nas  tne  contractor  certified  tnat  tne  qualification  of  tne 
CPCI  was  successfully  accomplished  in  accordance  with 
government-approved  CPCI  Test  Plans  and  Test  Procedures? 

<2.e) 

14)  Have  all  planned  objectives  for  wnich  test  results  (  )  (  ) 

differed  from  expected  results  beyond  specified  limits 

beeo  identified  with  actual  test  results  shown?  (2.e.l) 

15)  Have  any  planned  test  objectives  for  which  no  actual  (  )  (  ) 

results  were  obtaioed  been  identified  with  reasons  for 

not  fulfilling  such  objectives  being  stated?  (2.e.2) 

2 . f .  RECOMMENDATIONS 

16)  Have  recommendations  based  on  te3t  results  been  stated  (  )  (  ) 

for  subsequent  actions?  (2.f) 

17)  Are  all  recommendations  ooe  of  the  following  types:  (  )  (  ) 

a.  revision  of  the  CPCI  code  to  meet  specifically  identified 
requirements  which  were  not  fulfilled; 

b.  revision  of  the  Part  I  CPCI  Development  Specification  in 
cases  where  the  test  results  disclose  ambiguity  or  con¬ 
flicting  requirements ; 

c.  additional  testing  to  fulfill  objectives  for  which  results 
were  not  as  expected;  or 

d.  qualification  of  those  functions  for  which  test  objectives 
have  beeo  fulfilled.  (2.f) 

2.g.  TEST  CONDUCT  LOG 

18)  Has  a  record  of  each  test  been  prepared?  (2.g)  (  )  (  ) 

19)  Was  a  test  record  for  each  test  maintained  during  the  (  )  (  ) 

conduct  of  tne  test?  (2.g) 

20)  Was  tne  test  record  for  each  test  submitted  with  the  (  )  (  ) 

Test  Report?  (2.g) 

21)  Do  tne  test  records  for  each  test  documeot  actually  test  (  )  (  ) 

progress?  (2.g) 

22)  Are  tne  test  records  for  each  test  directly  traceable  (  )  (  ) 

to  the  CPCI  DT&E  Test  Procedures?  (2.g) 
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Yes  No 

2 3)  Does  the  test  record  for  each  test  contain  the  problems  (  )  (  ) 

encountered?  (2.g) 

24)  Does  the  test  record  for  each  test  contain  the  number  of  (  )  (  ) 

times  that  individual  steps  were  attempted?  (2.g) 

25)  Does  the  test  record  for  each  test  contain  the  backup  (  )  (  ) 

points  (i.e.,  steps  where  tests  resumed  retesting  fixes)? 

(2.g) 

26)  Does  the  test  record  for  each  test  indicate  any  impacts  (  )  (  ) 

of  computer  program  fixes  on  operator  procedures,  outputs 

(oew,  modified,  deleted)  or  equipment  functions  (alarms, 
lights,  switches)?  (2. g) 

2.h.  TROUBLE  REPORT  LOG 

27)  Has  a  record  of  all  CPCI  malfunctions  been  prepared?  (  )  (  ) 

(2.h) 

28)  Was  the  CPCI  malfunction  record  maintained  during  the  (  )  (  ) 

conduct  of  the  test?  (2.h) 

29)  Was  the  record  of  CPCI  malfunctions  submitted  with  the  (  )  (  ) 

Test  Report?  (2.h) 

30)  Does  the  record  for  each  CPCI  malfunction  include  a  (  )  (  ) 

computer  program  problem  identification?  (2.h) 

3D  Does  the  record  for  each  CPCI  malfunction  include  a  test  (  )  (  ) 

procedure  step  number  where  the  problem  was  detected? 

(2-h) 

32)  Does  the  record  for  each  CPCI  malfunction  Include  the  (  )  (  ) 

resolution  D.e.,  computer  program  fix)  closing  the 

problem?  (2.h) 

33)  Is  the  record  of  CPCI  malfunctions  traceable  to  the  (  )  (  ) 

test  conduct  log?  (2.h) 
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Section  2.  Teat  PlanniOK  Working  Group  (TPWG) 

2-1.  Review  AFR  80-14/AFSC  Sup  1,  Para  13  for  TPWG  compliance  requirements. 
2-2.  Review  AFSC  DK  4-2,  Chapter  5,  for  counter  program  testing  requirements. 
2-3.  Assist  the  TPWG  io  planning  for  software  testing. 

2-4.  Coordinate  software  test  Issues  with  users  and  support  agencies. 

2-5.  Monitor  and  oontrol  all  software  action  items  generated  by  the  TPWG. 

2-6.  Review  TEMP  when  it's  updated. 

2-7.  Review  Test  Plaos/Procedures. 
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Section 


Test  and  Evaluation  Master  Plan  (TEMP) 


3-1.  Assist  with  draft  TEMP  to  insure  it  meets  AFR  800-1**,  Vol  II  (Chapter 
5),  and  AFR  80-14,  AFSC  Support  Test  and  Evaluation.  (Paragraphs  8  &  14) 

3-2.  Insure  that  software  testing  is  adeauately  planned  in  the  TEMP  to 
include: 

a.  Addressing  S/W  critical  questions. 

b.  Areas  of  risk. 

c.  S/W  test  objectives. 

d.  S/W  responsibilities  of  all  participants. 

e.  S/W  schedules. 

f.  S/W  needed  test  resources. 

3-3.  Assist  in  determining  software  test  support  items  and  facilities. 

3-4.  Coordinate  TEMP  with  Operating  Command,  Supporting  Command  plus  any 
other  agency  involved  in  T*E  (i.e.,  AFTEC). 

NOTE:  The  TEMP  may  include  hardware  information  as  well  as  software 
ioformatioo. 
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Section  4.  Test  Plana 

4-1.  Review  AFR  800-14,  Volume  II,  Cnapter  5,  Teat  Plan/Procedures  Compliance 
Requirements. 

4-2.  Review  AFR  8C0-14  for  all  test  and  evaluation  requirements. 

4-3-  Review  the  DCP  for  critical  questions  and  Issues  whicn  must  be  addressed 
by  tne  total  system  test  program. 

4-4.  Review  tne  Computer  Program  Configuration  Item  Test  Plans  Checklist, 
Figure  IV-4-1. 

4-5.  Review  AFSC  DH4-2;  Design  Notes  DN5A3;  DN5A4,  DN5C1 ,  DN5C2,  DN5C3 , 

DN5D2,  DN5D3,  aod  DN5D4.  (Design  ootes  are  found  in  TOET.) 

4-6.  Review  ESD-TR-77-263.  Verification  Para  2.3*3  for  verifying  the  CPCI 
DT&E  plan. 

4-7.  Review  ESD-TR-77-255,  Software  Quality  Assurance  Para  3.4.4  for 
compliance  requirements. 

4-8.  Review  DID  DI-T-3703A  for  Test  Planning  Information. 

4-9*  Review  documents  submitted  against  respective  data  item  descriptions  to 
ascertain  adequacy. 

4-10.  Insure  that  all  functional  aod  performance  requirements  of  the 
appropriate  B-5  specifications  are  treated  in  the  CPCI  Test  Plan,  aod  specific 
test  objectives  are  assigned  to  each  test. 

4-11.  Insure  that  the  mechanism  for  conducting  each  test,  individual 
responsibilities  and  deficiency  reporting  process  are  described  in  the 
appropriate  CPCI  Test  Plan. 
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Figure  IV-4-1  COMPUTER  PROGRAM  CONFIGURATION  l  I  KM  iCPCi)  JEST  PLANS  ’irO::.';.: 

A  draft  revision  of  the  Development  Test  and  Evaluation  (DT&RI  CPCI  lost  ; ; an 
is  normally  submitted  after  authentication  of  each  CPC  is  Computer  Pro/rum 
Development  Specification  (CPDS)  and  before  Preliminary  Design  Review.  It  in  based 
on  Section  4  of  the  CPDS.  The  Data  Item  Description  Computer  Program  Configuration 
Item  (CPCI)  Test  Plan  recommends  tnat  approval  of  the  CPCI  Test  Plan  not  be  delayed 
beyond  Critical  Design  Review.  This  Data  Item  Description  also  recommends  ‘ ha* 
(witn  the  exception  of  a  Verification  Cross  Reference  index  iVCRl)  in  Section  '  of 
the  CPDS)  all  test  requirement  information  including  tne  method  of  verification 
included  in  the  CPCI  Test  Plan.  The  Test  Plan  should  also  be  submitted  as  a  single 
data  item  including  information  for  both  Preliminary  qualification  Test  ana  Parma  1 
Qualification  Test. 

This  checklist  for  reviewing  a  DT&E  CPCI  Test  Piari  is  basea  on  the  requirements 
stated  in  Data  Item  Description  D1-T-370UA  Computer  Program  Configuration  Item 
(CPCI)  Test  Plans/Procedures  0T-Dl-E-q01^>4.  In  reviewing  a  CPCI  test  plan,  the 
reviewer  must  also  be  aware  of  any  supplemental  or  contrary  direction  for 
preparation  of  tne  computer  program  test  plan  contained  in  the  Statement  of  Work 
(SOW),  in  the  Contract  Data  Requirements  List  ( CDRL)  or  in  CDRL  backup  sheets. 

The  checklist  questions  are  numbered  sequentially  accorumg  to  the  preparation 
Instructions  contained  in  the  cited  Data  Item  Description.  The  parenthetical 
numbers  following  the  questions  refer  to  the  related  paragraphs  of  the  instructions 
in  the  DID  Computer  Program  Configuration  Item  (CPCI)  Test  Plan. 

NOTE:  Whenever  a  "no"  answer  appears,  exceptions  should  be  listed  with 
accompanying  explanations  on  the  attached  backup  sheets. 

A.  PURPOSE  Yes  No 

(1)  Is  the  CPCI  identified  by  approved  number  and  nomenclature?  (  )  (  ) 

(a. ) 

(2)  Is  the  purpose  of  tne  Development  Test  and  Evaluation  ( DT&E)  (  )  (  ) 

CPCI  Test  Plan  stated  in  terms  of  establishing  detailed 

requirements,  criteria,  general  methods,  responsibilities,  and 
overall  planning  to  confirm  that  the  requirements  of  Section  3 
of  the  Part  I  CPCI  Specification  (CPDS)  are  verified  in  accordance 
with  Section  4  of  the  CPDS?  (a.) 

(3)  If  only  portions  of  the  CPCI  are  being  tested,  are  those  (  )  (  ) 

portions  correctly  designated?  (a.) 

(4)  Have  reasons  for  excluding  any  portion  of  the  CPCI  from  (  )  (  ) 

test  been  stated?  (a.) 
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d.  REFERENCES  Yes 

(6)  tiave  ail  documents  basic  to  or  significantly  related  to  (  ) 

tne  DT&E  CPCI  'lest  Plan  been  listed'/  (b.) 

1 o )  have  ail  documents  defining  the  CPCI  configuration  to  which  (  ) 

the  test  plan  applies  specifically  been  identified?  (b.) 

C.  TbSi’  CONCEPTS 

(7)  Is  there  sufficient  background  information  to  substantiate  (  ) 

the  test  philosophy?  (c.) 

id)  is  there  sufficient  information  (e.g.,  definition  of  test  (  ) 


objectives  and  approach)  for  understanding  and  evaluating  the 
lest  Plan?  (c . ) 

D.~”  cUrtulFlCATlON  REQUIREMENTS  AND  CRITERIA 

D. 1  Detailed  Qualification  Requirements 

(9)  Has  all  detailed  qualification  information  or  test  objectives  (  ) 

based  on  each  of  the  functional  and  performance  requirements  of 
Section  3  of  tne  CPDS  been  presented  (or  attached  in  appendices 

or  additional  volumes)?  (d.l) 

D. 2  wualification  Conditions 

(10)  Have  all  conditions  under  which  qualification  of  each  (  ) 

functional  and  performance  requirement  must  be  achieved  been 
specified  in  terms  of  the  ranges  for  input  values,  specific 
initial  values,  and  amount  of  different  types  of  input  data? 

Id.  2} 

(11)  Do  these  test  conditions  describe  each  CPCI  parameter  which  (  ) 

must  be  tailored  to  accomplish  specified  test  objectives?  (d.2) 

D. 3  Acceptance  Criteria 

(12)  Have  acceptance  criteria  for  each  identified  performance  (  ) 

and  functional  requirement  been  stated  in  terms  of  presence  or 
aosence  of  specified  outputs  and  tolerance  limits  for  calculated 
values?  (d.3) 

(13)  Are  all  acceptance  criteria  traceable  to  specific  (  ) 

test  objectives?  (d. j) 

D. 4  Performance  Methods 

(14j  iiave  ail  methods  (including,  for  example,  analysis  of  data,  (  ) 

and  examination  of  displays,  and  equipment  response  to  computer 
program  operation)  for  determining  correspondence  of  program 
performance  to  these  parameters  been  stated  for  each  performance 
and  functional  requirement';  (d.4) 


No 

( 

( 


( 

( 


( 


( 


( 


( 


( 
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E.  QUALIFICATION  OBJECTIVES/TEST  PHASE  SUMMARY 


Yea  No 


(15)  Does  the  Test  Plan  indicate  that  some  qualification  testing  (  )  ( 

must  occur  prior  to  Formal  Qualification  Testing  ( FQT)?  (e.) 

E.l  Computer  Programming  Test  and  Evaluation  Requirements 

(16)  Have  all  functional  and  performance  requirements  against  (  )  ( 

which  the  operation  of  selected  modules  are  to  be  qualified 

during  Computer  Programming  Test  and  Evaluation  (CPT&E)  been 
listed?  (e.l) 

E.2  Preliminary  Qualification  Testing 

(17)  Have  all  functional  and  performance  requirements  against  (  )  ( 

which  the  Computer  Program  Components  (CPCs)  are  to  be  qualified 
during  Preliminary  Qualification  Test  (PQT)  been  listed?  (e.2) 

E-3  Formal  Qualification  Testing  Requirements 

(18)  Have  all  the  functional  and  performance  requirements  against  (  )  ( 

which  the  operation  of  the  CPCI  is  to  be  qualified  during 

Formal  Qualification  Testing  (FQT)  been  listed?  (e.3) 


E.4  System  DT&E  Requirements 

(19)  Have  all  the  functional  and  performance  requirements  against  (  )  ( 

which  the  operation  of  the  CPCI  is  to  be  Qualified  but  must  be 
deferred  until  system  DT&E  testing  been  listed  and  included  in 
the  system  DT&E  Test  Plan?  (e.4) 


F.  DT&E  CPCI  QUALIFICATION  TEST  IMPLEMENTATION 


F. 1  Location  and  Schedule 

(20)  Has  the  location  at  which  the  qualification  tests  will  be  (  )  ( 

conducted  been  specified?  (f.l) 

(21)  Has  the  schedule  for  the  tests  been  established  in  terms  of  (  )  ( 

one  or  more  of  following:  dates  for  particular  tests  or 

sets  of  tests;  general  periods  (weeks  or  months)  for  various 
tests  or  phases  of  testing;  and  periods  relative  to  milestones 
in  overall  acquisition  schedule?  (f.l) 

(22)  If  the  CPCI  is  to  be  developed  incrementally,  is  the  PQT  (  )  ( 

schedule  concurrent  with  such  incremental  development?  (f.l) 

(23)  If  the  PQT  or  FQT  are  to  be  accomplished  incrementally  (in  (  )  ( 

phases),  are  specific  test  plan  paragraphs  to  be  included  with 

each  test  phase  defined?  (f.l) 
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F.2  Limitations  and  Geoeral  Comments 

(24)  Have  general  comments  relative  to  test  implementation  and 
accomplishment  of  test  objectives  been  included?  (f.2) 

(25)  Have  all  limitations  (if  any)  relative  to  test  implementation 
and  accomplishment  of  test  objectives  been  identified?  (f.2) 

F-3  Preparation  of  Inputs 

(26)  Have  methods  for  preparation  of  all  qualification 
test  data  been  defined?  (f.3) 

(27)  Have  the  simulation  and/or  test  case  generation  vehicles  to 
be  used  for  preparation  of  all  test  input  data  been  Identified? 

(f.3) 

(28)  Have  requirements  for  review  or  validation  of  all  test  input  (  )  (  ) 

data  been  specified?  (f.3) 

(29)  Have  requirements  for  all  agencies  (other  than  the  contractor)  (  )  (  ) 
involved  in  preparation  of  test  input  data  been  specifically 
identified?  (f.3) 

(30)  Have  responsibilities  for  these  other  agencies  been  (  )  (  ) 

specifically  defined?  (f.3) 

F . 4  Cooduct  of  the  Tests 

(31)  Have  geoeral  procedures  for  test  cooduct  been  established?  (  )  (  ) 

(f  .4) 

(32)  Have  responsibilities  for  test  direction,  operation,  (  )  (  ) 

and  observation  been  delineated?  (f.4) 

(33)  Has  a  test  conduct  log  to  record  actual  test  events  (  )  (  ) 

beeo  indicated?  (f.4) 

(34)  Have  requirements  for  pre-test  briefings  and  (  )  (  ) 

post-test  debriefings  beeo  identified?  (f.4) 

F.5  Analysis  of  Results 

(35)  Have  geoeral  procedures  for  analysis  of  qualification  (  )  (  ) 

test  results  beeo  described?  (f.5) 

(3 6>  Have  all  compuer  programs  to  be  used  for  data  (  )  (  ) 

reduction  and  analysis  beeo  identified?  (f.5) 

(37)  Have  all  requirements  and  responsibilities  of  agencies  other  (  )  (  ) 

thao  the  contractor  beeo  specifically  identified?  (f.5) 


Yes  No 
(  )  (  ) 

(  )  (  ) 

(  )  (  ) 

(  )  (  ) 
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F.6  Equipment  and  Computer  Program  Requirements  Yes  No 

(38)  Have  requirements  for  all  computer  programs  (other  than  the  (  )  (  ) 

CPCI  being  tested)  involved  in  CPCI  qualification  testing  been 
summarized  (f.6) 

(39)  Have  requirements  for  all  support  equipment  to  be  used  on  (  )  (  ) 

CPCI  qualification  testing  been  summarized?  (f.6) 

F.7  Personnel  Requirements 

(40)  Have  the  personnel  requirements  for  each  agency  or  contractor  (  )  (  ) 

or  contractor  involved  in  qualification  testing  been 

summarized?  (f.7) 

(41)  Have  the  personnel  requirements  for  each  agency  or  contractor  (  )  (  ) 

involved  in  testing  been  specified  in  terms  of  the  following? 

(f.7) 


Responsibility 

Authority 

Know ledge/Sk il Is 

Defined? 

Defined? 

Defined? 

Test  Agency  1# 

Yes  No 

Yes  No 

Yes  No 

Position  l  (  )  (  )  (  )  (  )  (  )  (  ) 

Position  2  (  )  (  )  (  )  (  )  (  )  (  ) 

Position  3* _ (  )  (  )  (  )  (  ) _ (  )  (  ) _ 

#  Repeat  this  table  on  backup  sheets  for  each  agency  or  contractor  and  each 
personnel  position. 


Yes  No 

(42)  Has  the  organization  responsible  for  the  conduct  of  (  )  (  ) 

testing  been  described?  (f.7) 

G.  CONTROL  AND  REPORTING  PROCEDURES.  Control  procedures  and  documentation  may  be 
specified  by  reference  to  existing  procedures  or  requirements  and  by  specific 
identification  of  necessary  exceptions  or  changes.  Such  references  shall  be 
made  only  to  other  contractually  delivered  documents,  such  as  the  Configuration 
Management  Plan  or  the  Computer  Program  Development  Plan. 

G.l  Control  of  the  DT&E  CPCI  Test  Program 

(43)  Have  procedures  for  revising  or  updating  the  DTAE  CPCI  Test  (  )  (  ) 

Plao  as  a  result  of  schedule  changes,  changes  to  design 
requirements  or  CPCI  detail  design,  revised  provisions  for 

supporting  the  test  program,  etc.,  been  specified?  (g.l) 


(44)  Have  provisions  for  retesting  due  to  design  changes  or 

correction  of  errors  found  in  earlier  testing  been  included? 


(  )  (  ) 
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Yes  No 

(45)  Have  requirements  and  procedures  for  controlling  the  (  )  (  ) 

ioterrelatioo  between  the  DT4E  CPCI  Test  Program  and 

configuration  control  of  the  CPCI  design  requirements  been 
established?  (g.l) 

G.2  Documentation  of  Test  Procedures 

(46)  Have  provisions  for  maintaining  traceability  between  (  )  (  ) 

corresponding  CPCI  Test  Plans  and  Test  Procedures  been 

specified?  (g.2) 

(47)  Have  provisions  for  maintaining  currency  between  the  (  )  (  ) 

corresponding  CPCI  Test  Plan  and  Test  Procedures  described? 

(g.2) 

(48)  Has  the  approacn  for  handling  last  minute  redline  cnaoges  (  )  (  ) 

(i.e.,  changes  generated  immediately  before,  or  during,  any 

test)  to  the  Test  Procedures  been  described?  (g.2) 


Documentation  of  Test  Reports 


(49)  Have  provisions  for  preparing  reports  of  individual 
qualification  tests  beeo  specified?  (g . 3 ) 


)  (  ) 


(50)  Have  provisions  for  reviewing  reports  of  individual  (  )  (  ) 

qualification  tests  beeo  specified?  (g-3) 


(51)  Have  provisions  for  preparing  other  reports  which  may  be  (  )  (  ) 

related  to  the  DT4E  CPCI  Test  Program  been  specified?  (g.3) 


(52)  Have  provisions  for  reviewing  other  reports  which  may  be  (  )  (  ) 

related  to  the  DT4E  CPCI  Test  Program  been  specified?  (g.3) 


(53)  Have  provisions  for  preparing  separate  test  reports  for  PQT  (  )  (  ) 

and  FQT,  as  well  as  for  any  increment  or  test  phase  beeo 
described?  (g.3) 


>  w. 


HELPi),  Volume  IV,  Revision  I  1  Oecemner  1 

Test  and  Independent  Verification  and  Validation 


Section  5.  Test  Procedures 

5-1.  Review  GT-DI-E-30153,  Computer  Program  Configuration  item  CPCi  Test 

Procedures. 

5-2.  Review  ESD-TR-77-2P 3 .  Verification  Para  3.2.3  for  verifying  tne  CPCI  DT%E 

procedures . 

5-3-  Insure  tnat  all  step-ny-step  procedures,  specific  success  criteria  and  da 
reduction/analysis  techniques  arid  results  are  given  for  each  test  case,  in  the 
appropriate  CPCI  Test  Procedures. 

5-4.  Coordinate  with  contractor  to  resolve  document  deficiencies. 

5-5.  Review  all  changes  to  the  documents  as  a  result  of  Government  comments. 

5-6.  Review  Computer  Program  Conf iguration  Item  (CPCI)  Test  Procedures  Cneov.li 

Figure  IV-5-1. 
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figure  XV— COMPUTER  PROGRAM  CONFIGURATION  ITEM  (CPCI)  TEST  PROCEDURES  CHECKLIST 

i:iu>  cnecKilst  for  reviewing  CPCI  Test  Procedures  is  based  on  the  requirements 
stated  in  Gl’-uI-E-^Olo**,  Computer  Program  Configuration  Item  (CPCI  Test 
Procedures.  In  reviewing  CPCI  test  procedures,  the  reviewer  must  also  be  aware  of 
any  supplemental  or  contrary  direction  for  preparation  of  the  computer  test 
procedures  contained  in  tne  Statement  of  Work  (SOW),  in  the  Contract  Data 
Requirements  List  (CDRL)  or  in  CDKL  backup  sneets. 

The  Development  Test  and  Evaluation  (DT&E')  CPCI  Test  Procedures  are  normally 
submitted  after  Government  approval  of  the  corresponding  DT&E  CPCI  Test  Plan  and 
should  De  directly  traceable  to  this  Test  Plan.  It  is  recommended  that  a  separate 
data  item  submission  of  Test  Procedures  De  accomplished  for  Preliminary 
Qualification  Testing  and  Formal  Qualification  Testing. 

The  DID  requires  the  contractor  to  prepare  a  separate  DT&E  Test  Procedure  for 
eacn  individual  DT&E  qualification  test.  A  copy  of  this  checklist  should  be 
completed  for  eacn  such  DT&E  Test  Procedure.  However,  question  (1),  below,  need  be 
completed  for  only  the  f Lrst  of  these. 

Tne  checklist  questions  are  numbered  sequentially  according  to  the  preparation 
instructions  contained  in  tne  cited  Data  Item  Description.  The  parenthetical 
numbers  following  tne  questions  refer  to  the  related  paragraphs  of  the  instructions 
in  OT-DI-E-3015R,  Computer  Program  Configuration  Item  (CPCI)  Test  Procedures. 

Whenever  a  "no"  answer  appears,  exceptions  should  be  listed  with  accompanying 
explanations  on  the  attacned  backup  sheets. 

GENERAL  Yes  No 

(1)  is  there  a  Development  Test  and  Evaluation  (DT&E)  Test  (  )  (  ) 

Procedure  for  each  individual  DT&E  Computer  Program 
Configuration  Item  (CPCI)  qualification  test?  (10) 

A.  CAPTION 

(E)  Does  the  test  procedure  include  an  identifying  caption?  (a.)  (  )  (  ) 

A . i  Test  Identification 

(3)  Is  tne  individual  test  for  wnich  the  test  procedure  is  written  (  )  (  ) 

uniquely  identified  in  that  test  procedure's  caption?  (a.l) 

(4)  Does  tne  caption  indicate  whether  the  test  procedure  is  (  )  (  ) 

for  a  Preliminary  or  Formal  Qualification  Test?  (a.l) 

A./  Contract  item 


(b)  Does  tne  caption  contain  the  identification  number  of  the 
CFCl  to  which  the  test  applies?  (a. 2) 


(  )  (  ) 
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Yes 

(6)  Has  the  approved  nomenclature  for  the  CPCI  to  which  the  (  ) 

test  applies  been  specified  in  the  caption?  (a. 2) 

A.  3  Primary  Function 

(7)  Have  all  the  CPCI's  primary  functions  or  segments  to  be  (  ) 

tested  been  identified  in  the  caption?  (a. 3) 

B.  LOCATION  AND  SCHEDULE 


(8)  Have  the  location  and  schedule  for  briefings  been  (  ) 

specified?  (b.l) 

(9)  Have  the  location  and  schedule  for  tests  been  specified?  (  ) 

(b.2) 

(10)  Have  the  location  and  schedule  for  debriefings  been  (  ) 

specified?  (b.3) 

(11)  Have  the  location  and  schedule  for  data  reduction  and  (  ) 

analysis  been  specified?  (b.4) 


C.  REFERENCES 


(12)  Is  there  a  reference  to  the  DT4E  CPCI  Test  Plan?  (c.l)  (  ) 

(13)  Are  there  references  to  the  CPCI  Computer  Program  (  ) 

Development  Specifications  for  each  of  the  functions 

to  be  tested?  (c.2) 

(14)  Are  there  references  to  appropriate  users'  manuals  or  (  ) 

positional  handbooks  for  the  CPCI  being  tested?  (c.3) 

(15)  Are  there  references  to  users'  manuals  for  the  test  or  (  ) 

support  computer  programs?  (c.4) 

D.  TEST  OBJECTIVES 


(16)  Have  the  detailed  test  objectives  been  specified  by  functional  (  ) 

description  and  references  to  the  Qualification  Requirements  and 
Criteria  section  (see  the  DID  on  the  CPCI  Test  Plan,  Section  d) 

of  the  DT4E  CPCI  Test  Plan?  (d.) 

E.  MANNING  AND  RESPONSIBILITIES 

(17)  Have  requirements  and  responsibilities  for  all  essential  test  (  ) 

personnel  (including  console  operators,  test  directors, 
technical  consultants,  data  analysts,  and  any  others )  been 
specified?  (e.) 

(18)  Have  special  knowledge  and  skills  requirements  been  stated  (  ) 

for  all  test  personnel  needing  them?  (e.) 


No 
(  ) 


(  ) 


(  ) 

(  ) 

(  ) 

(  ) 


(  ) 
(  ) 


(  ) 

(  ) 


(  ) 


(  ) 


(  ) 
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(19)  Have  individual  contractors  or  agencies  responsible  for 
supplying  personnel  been  identified?  (e.) 

(20)  Have  the  responsibilities  for  each  3uch  individual 
contractor  or  agency  been  defined?  (e.) 

(21)  Have  personnel  requirements  identical  to  those  stated 

in  tne  DTAE  CPCI  Qualification  Test  Implementation  section  (see 
the  DID  on  the  CPCI  Test  Plan,  Section  f)  of  the  DTAE  Test  Plan 
beeo  indicated  by  reference  to  the  test  plan?  (e.) 

F.  EQUIPMENT  AND  COMPUTER  PROGRAM  REQUIREMENTS 

(22)  Have  requirements  for  all  computer  programs  needed  for  the 
test,  other  than  the  CPCI  being  tested,  been  specified?  (f.) 

(23)  Have  requirements  for  all  equipment  necessary  to 
support  the  test  been  specified?  (f.) 

(24)  Have  any  equipment  and  computer  program  requirements  ' 
identical  to  those  stated  in  the  DTAE  CPCI  Qualification  Test 
Implementation  section  (see  the  DID  on  the  CPCI  Test  Plan, 
Section  f)  of  the  DTAE  CPCI  Test  Plan  been  specified  by 
reference  to  the  test  plan?  (f.) 

G.  TEST  OPERATING  PROCEDURES 

G. 1  Computer  Program  Initiation 

(2D)  Have  procedures  for  reading  the  program  to  be  tested  into 
one  computer  been  specified?  (g.l) 

(26)  Have  procedures  for  establishing  the  operation  mode  required 
by  the  computer  program  being  tested  beeo  specified  (g.l) 

(27)  Have  procedures  for  initializing  the  parameters  required 
been  specified?  (g.l) 

(28)  Have  procedures  for  providing  the  inputs  required  beeo 
specified?  (g.l) 

(29)  Have  procedures  for  collecting,  storing  aod  displaying 
the  outputs  produced  been  specified?  (g.l) 

' 30 )  Have  procedures  for  beginning  operation  of  the  computer 
program  being  tested  been  specified?  (g.l) 

(3U  Have  listings  of  tne  input  material  (e.g.,  card  decKs) 

to  accomplish  the  initiation  of  computer  program  operation 
been  provided  as  an  appendix?  (g.l) 


Ye  3  No 
(  )  (  ) 

(  )  (  ) 

(  )  (  ) 


(  )  (  ) 

(  )  (  ) 

(  )  (  ) 


(  )  (  ) 

(  )  (  ) 

(  )  (  ) 

(  )  (  ) 

(  )  (  ) 

(  )  (  ) 

<  M  ) 
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Yes  No 

G.2  Maintaining  Computer  Program  Operation 

(32)  Have  all  procedures  for  operator  intervention  (such  as  (  )  (  ) 

maintaining  input  data  flow  and  replenishing  card  and  tape 

supplies)  required  to  maintain  program  operation  been 
specified?  (g.2) 

G. 3  Computer  Program  Termination  and  Restart 

(33)  Have  adequate  procedures  for  normal  termination  of  the  (  )  (  ) 

computer  program  being  tested  been  specified?  (g.3) 

(3*0  Have  satisfactory  procedures  for  unscheduled  termination  (  )  (  ) 

of  the  computer  program  being  tested  been  specified?  (g.3) 

(35)  Have  adequate  procedures  for  restarting  program  operation  (  )  (  ) 

to  iosure  that  necessary  output  data  shall  be  obtained 

been  specified?  (g*3) 

H.  DETAILED  TEST  DESCRIPTION 
H. 1  Test  Results 

(36)  Has  a  detailed  description  of  all  test  outputs  been  (  )  (  ) 

given?  (h.l) 

(37)  Has  a  detailed  description  of  all  unexpected  test  events  (  )  (  ) 

been  given?  (h.l) 

(38)  Has  a  detailed  description  of  all  expected  test  results  (  )  (  ) 

been  given?  (h.l) 

(39)  Have  all  test  objectives  satisfied  or  partially  satisfied  by  (  )  (  ) 

each  test  result  been  identified  by  reference  to  the 

Qualification  Requirements  and  Criteria  section  (see  the  DID  on 
the  CPCI  Test  Plan,  Section  d)  of  the  DT&E  CPCI  Test  Plan?  (h.l) 

(40)  Have  all  test  events  been  described  in  the  order  in  which  (  )  (  ) 

they  are  planned  to  occur?  (h.l) 

(41)  Have  all  dependencies  among  test  events  been  indicated?  (  )  (  ) 

(h.l) 

(42)  If  more  than  one  operating  or  monitoring  position  is  (  )  (  ) 

involved,  has  the  sequence  of  events  for  each  position 

been  indicated?  (h.l) 

(43)  Has  the  interdependence  of  operating  positions  with  (  )  (  ) 

respect  to  specific  test  events  beeo  described?  (h.l) 
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Yes  No 


H. 2  Test  Inputs 

(44)  Have  tne  listings  of  test  Inputs  or  listings  produced  in  the  (  )  (  ) 

preparation  of  test  inputs  (e.g.,  execution  of  a  test  case 
geoeratioo  program)  been  included  in  an  appendix?  (h.2) 

I.  DATA  REDUCTION  AND  ANALYSIS 

1.1  Recording  and  Reduction  Requirements 

(45)  Have  all  data  which  must  be  recorded  during  the  test  by  (  )  (  ) 

program,  manually,  or  by  instrumentation  been  specified?  (i.l) 

(46)  Have  formatting  requirements  for  ail  the  data  resulting  from  (  )  (  ) 

the  reduction  and  analysis  processes  been  specified?  (i.l) 

(47)  Have  the  conteot  requirements  for  all  the  data  resulting  from  (  )  (  ) 

the  reduction  and  analysis  processes  been  specified?  (i.l) 

(48)  Have  the  requirements  for  data  recording  and  reduction  been  (  )  (  ) 

specified  in  sufficient  detail  and  in  a  manner  such  that  the 
resulting  information  will  clearly  show  wnether  the  test 

objectives  have  been  met?  (i.l) 

1.2  Data  Reduction  Analysis  Procedures 

1. 2.  a  General 

(49)  Have  the  procedures  to  be  employed  in  reducing  and  analyzing  (  )  (  ) 

test  data  been  specified  by  references  and  exceptions  to  the 

CPCI  Test  Procedure's  References  section  (see  DID  Section  c)? 

(i.2.a) 

1.2. b  Computer  Data  Reduction  and  Analysis 

(50)  Has  tne  data  reduction  and  analysis  to  be  accomplished  (  )  (  ) 

by  computer  programs  been  identified?  (i.2.b) 

1 . 2 .  b . 1  Computer  Program  Initiation 

(51 )  Have  acceptable  procedures  to  read  the  data  reduction  and  (  )  (  ) 

analysis  program  into  the  computer  been  specified?  (i.2.b.l) 

(52)  Have  satisfactory  procedures  to  establish  the  mode  of  (  )  (  ) 

operation  required  by  the  data  reduction  and  analysis 

program  been  specified?  (i.2.b.l) 

(53)  Have  reliable  procedures  to  initialize  the  parameters  (  )  (  ) 

required  by  the  data  reduction  and  analysis  program 

beeo  specified?  (i.2.b.l) 
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(54)  Have  acceptable  procedures  to  provide  the  inputs  required 
by  the  data  reduction  and  analysis  program  been  specified? 
(i.2.b.l) 

(55)  Have  adequate  procedures  to  collect,  store  and  display 
the  outputs  required  by  the  data  reduction  and  analysis 
program  beeo  specified?  (i.2.b.l) 

(56)  Have  satisfactory  procedures  for  beginning  operation  of 
the  data  reduction  and  analysis  computer  program  beeo 
specified?  (i.2.b.l) 

(57)  Have  listings  of  all  input  materials  (e.g.,  card  decks, 
tapes)  for  the  above  initialization  of  computer  programs 
beeo  provided  in  an  appeodix?  (i.2.b.l) 

1.2. b. 2  Maioteoaoce  of  Computer  Program  Operation 

(58)  Have  procedures  for  operator  intervention  (such  as 
maintaining  input  data  flow  and  replenishing  card 

and  tape  supplies)  required  to  maintain  computer  program 
operation  been  specified?  (i.2.b.2) 


Yes  No 
(  )  (  ) 

(  )  (  ) 

(  )  (  ) 

(  )  (  ) 

(  )  (  ) 


I.2.D.3  Computer  Program  Termination  and  Restart 

(59)  Have  adequate  procedures  for  normal  termination  of  the  data  (  )  (  ) 

reduction  and  analysis  program  beeo  specified?  (i.2.b.3) 

(60)  Have  satisfactory  procedures  for  unscheduled  termination  of  the  (  )  (  ) 

data  reduction  and  analysis  program  been  specified?  (i.2.b.3) 

(61)  Have  adequate  restart  procedures  for  obtaining  necessary  (  )  (  ) 

output  from  the  data  reduction  and  analysis  program  beeo 

specified?  (i.2.b.3) 


I.2.C  Manual  Data  Reduction  and  Analysis 

(62)  Has  any  data  reduction  and  analysis  to  be  accomplished  (  )  (  ) 

manually  beeo  specified?  (i.2.c) 

(63)  Have  adequate  procedures  for  accomplishing  any  manual  data  (  )  (  ) 

reduction  and  analysis  been  established?  (i.2.c) 
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Section  6.  Preliminary  Qualification  Testing  ( PQT) 

6-1.  PQT  Preparation. 

A.  Review  Section  4  of  tne  Development  Specification  to  understand  the 
testing  responsibilities  of  the  contractor. 

B.  Review  AFR  800-14,  Vol  II,  Chapter  5. 

C.  Review  ESD-TR-77-327,  Software  Maintenance,  Formal  Qualification  Test, 
Para  2.2.4,  Pgs  24-25. 

D.  Review  and  approve  the  appropriate  CPCI  Test  Plan  and  Test  Procedures, 
making  sure  that  explicit  test  objectives  and  success  criteria  are  described  for 
each  test  case;  and  that  the  mechanism  for  conducting  the  test  and  resolving 
discrepancies  is  understood  by  all  participants. 

6-2.  PQT  Conduct. 

A.  Answer  the  following  questions  with  respect  to  software  maintenance 
implications : 

1)  Do  tne  test  procedures  call  for  adequate  inspection  of  the  specified 
maintainable  software  attributes  i.e,  module  size,  language,  structured  code, 
adherence  to  programming  standards,  code  reliability? 

2)  What  impacts  on  the  software  structure  have  design  changes, 
requirement  changes,  and  error  corrections  made? 

3)  Are  the  traceability  matrix  and  test  procedures  curreot  with  design 
and  requirement  changes? 

4)  Have  time  sensitive  portions  of  the  software  been  adeouately 
identified  and  documented  for  maintenance  programming? 

5)  Are  the  listings: 

a.  readable? 

b.  reasonably  self-documented? 

c.  adequately  commented? 

d.  easily  reviewable? 

e.  clear  as  to  what  each  area  of  code  is  intended  to  do? 

6)  Are  the  data  for  references  symbolic,  and  are  they  meaningful? 

7)  Are  the  date  and  version  of  the  listing  compatible  with  the 
contractor's  list  of  materials  to  be  qualified? 

8)  Are  all  development  and  test  tools  acceptable? 
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B.  Participate  in  the  PQT  Test  Conduct  Functions. 

6-3«  Post-Testing  Requirements. 

A.  Review  and  approve  the  CPCI  Test  Report  pertinent  to  POT,  if 
applicable. 

B.  Mooitor  resolution  of  program  trouble  reports  opened  during  the 
testing. 
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Section  7.  Formal  Qualification  Testing  (FQT) 

7-1.  Pre-Testing  Requirements 

A.  Review  ESD-TR-75-85,  Monitoring  and  Reporting  Software  Development 
Status,  Para  2.2.1,  Pg  31  for  a  comprehensive  list  of  what  is  expected  from 
tne  contractor. 

B.  Review  AFR  800-14,  Vol  II,  Chapter  5. 

C.  Review  appropriate  CPCI  Test  Plans  and  Test  Procedures  maxing  sure 
tnat  explicit  test  objectives  and  success  criteria  are  described  for  each  test 
case;  that  tne  mechanism  for  conducting  the  test  and  resolviog  discrepancies 
is  understood  by  all  test  participants;  and  specification  will  be  tested. 

7-2.  FQT  Conduct 

A.  Participate  in  the  FQT  Test  Conduct  functions  as  a  member  of  the  PO 
Test  Team. 

B.  Note  all  program  discrepancies  and  problems. 

7-3.  Post-Test  Requirements 

A.  Review  and  approve  the  CPCI  Test  Report  pertinent  to  FQT. 

B.  Monitor  resolution  of  Program  Trouble  Reports  opened  during  the  test 
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Section  8.  System  Integration  Tests 


8-1.  Review  ESD-TR-77-254 ,  Configuration  Management,  Section  6,  for  control 
during  system  testing. 


8-2.  Review  Section  4.1.5  of  the  Development  Specification  for  an 
understanding  of  contractors  commitments. 


8-3.  Review  all  appropriate  Test  Plans  and  procedures  for  pertinent  CPC I 
qualification  information. 


8-4.  Maiotain  a  Government  test  log/notebook  of  significant  test  events  and 
countersigned  by  an  official  witness. 


8-5.  Review  the  hardware/software  baseline  and  document  any  changes  during 
testing. 


8-6.  Mooitor  and  analyze  any  discrepancy  reports  generated  during  testing. 


8-7.  Review  report. 
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Figure  IV-9-1.  Independent  Verification  and  Validation  Guide  (IV&V) 

A.  Hq  USAF/RD  and  Hq  USAF/LE  have  established  a  policy  (Figure  IV-9-2)  for 
tne  use  of  Iodepeodeot  Verification  and  Validation  (IV&V)  for  Embedded 
Computer  Systems  (ECS).  This  policy  will  be  incorporated  into  a  revision  to 
AFSC  Supplement  1  to  AFR  800-14,  Management  of  Computer  Resources  in  systems. 
All  ESD  Program  Offices  will  consider  the  use  of  IV&V  on  new  acquisitions  and 
for  retrofit  or  modification  of  existing  systems. 

B.  IV&V  can  span  the  whole  life  cycle,  performing  requirements  analysis, 
design  analysis,  code  inspection,  and  actual  testing  of  the  computer 
programs.  It  is  up  to  the  program  office  to  determine  the  level  of  IV&V  for 
their  particular  program.  Individual  Computer  Program  Configuration  Items 
(CPCIs)  might  require  a  different  level  of  IV&V  because  of  their  criticality. 
Again,  this  has  to  be  assessed  by  the  program  office.  The  following 
methodology  provides  a  guide  for  both. 

C.  This  methodology  is  based  on  the  Impact  software  errors  can  have  on  the 
system.  The  objective  is  to  help  you  arrive  at  a  good  estimate  of  the  total 
oeed  for  IV&V,  from  requirements  analysis  through  independent  testing  of  the 
code  itself. 

D.  The  first  and  most  important  step  of  the  methodology  requires  the 
determination  of  criticality  values  for  each  CPCI.  The  criticality  value  is 
obtained  by  multiplying  the  criticality  class  by  the  probability  of  occurrence 
for  each  deoisioo  factor.  (See  Table  IV-9-i  for  the  definitions  of  what  the 
different  criticality  classes  are  and  the  values  for  both  the  criticality 
class  and  the  probability  of  occurrence.)  The  decision  factors,  such  as 
equipment  malfunctions,  are  specific  to  individual  programs.  The  factors 
should  be  based  on  what  could  go  wrong  with  a  specific  CPCI  which  would  impact 
the  system. 

E.  Tne  final  step  consists  or  adding  up  the  criticality  values  for  each  CPCI 
and  dividing  by  the  number  of  factors  to  get  an  IV&V  value.  The  corresponding 
IV&V  level  is  obtained  from  Table  IV-9-3-  The  following  ESD  example  is 
provided  for  your  information.  (See  Table  IV-9-4.) 

F.  An  important  thing  to  remember  is  that  this  is  only  a  guide  for  applying 
IV&V.  It  is  ultimately  the  responsibility  of  the  program  manager  to  determine 
whether  the  IV&V  that  is  to  be  applied  is  adequate  or  not. 
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U 


CRITICALITY 

CLASS 


TaDle  IV-9-1.  Decision  Taple 


ASSIGNED 

VALUE 


PROBABILITY 

OF 

OCCURRENCE 


ASSIGNED 

VALUE 


Negligible 

Marginal 

Critical 

Catastropnic 


Iopoesible 

Improbable 

Probable 

Frequent 


Where: 


Negligible; 


Marginal: 


Failure  of  software  related  to  the  factor/subsystem  under 
evaluation  would  create  inconvenience,  rerun  of  batch 
programs,  minor  cost,  etc. 

Failure  of  software  related  to  the  fac tor /susbsy stem  under 
evaluation  would  create  degradation  of  secondary  missions, 
some  schedule  delay  and/or  software  cost  overrun,  etc. 


Critical:  Failure  of  software  related  to  the  factor /subsystem  under 

evaluation  would  create  degradation  of  the  operational 
mission  to  a  degree  that  the  program  manager  would  not  use 
the  system  if  forewarned,  damage  of  equipment/property, 
injury  to  personnel,  substantial  schedule  delay  and/or 
software  cost  overrun,  etc. 


Catastrophic:  Failure  of  software  related  to  the  factor /susbsystem  under 

evaluation  would  create  loss  of  life,  mission  failure, 
injury  to  personnel,  critical  equipment  loss,  excessive 
delay  and/or  cost  overrun. 
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Table  IV-9-2.  Possible/Factor /Subsystem  Criticality  Values 


Probability  of 

Occurrence 

Criticality  Class 

Negligible  Marginal 

Critical 

Catastrophic 

Impossible 

Improbable 

Probable 

Frequent 


Table  IV-9-3*  IV4V  Level  Selection  Chart 


IV4V  VALUE 


SUGGESTED  IV4V  LEVEL 


Level  C:  Constructively  critique  developer's  documentation.  Participate  in 
milestone  reviews.  Identify  critical  requirements  and  design 
problems  and  reocmmend  solutions.  Monitor  equipment. 

Level  B:  Same  effort  as  in  Level  C.  In  addition,  using  appropriate  tools  as 
necessary:  Analyze  selected  critical  functions.  Spot  check  design 
performance.  Conduct  limited  testing.  Evaluate  critical  development 
test  results.  Perform  selected  audits. 


Level  A:  Same  effort  as  Level  B:  In  addition,  using  appropriate  tools  as 

necessary:  Independently  analyze  requirements  and  design.  Re-derive 
key  algorithms.  Confirm  technical  adequacy.  Independently  test  and 
evaluate  operational  software.  Cooduct  stress  tests.  Conduct 
special  studies.  Support  Configuration  and  Data  Management. 
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Figure  IV-9-2  Independent  Verification  and  Validation  (IV4V) 
for  Embedded  Computer  Systems  (ECS) 

1.  Background :  The  IV4V  concept  was  originated  during  the  early  days  of  the 
Ballistic  Missile  and  Space  Systems  Division.  It  was  applied  to  missile 
systems*  embedded  computer  programs  that  were  involved  in  the  activation  and 
control  of  nuclear  weapons  and  in  the  launching  of  space  vehicles.  The 
purpose  of  the  IV4V,  conducted  by  the  independent  contractor,  was  to  insure 
that  tne  software  properly  performed  all  intended  functions  and,  or  equal 
importance,  that  it  performed  no  uointended  functions.  The  IV4V  concept 
proved  highly  successful  by  permitting  critical  embedded  computer  programs  to 
function  more  reliably.  Because  of  its  success,  the  IV4V  techniques  spread  to 
other  embedded  computer  applications. 

2.  Purpose  and  Objective:  This  letter  establishes  Air  Force  policy  for 
Independent  Verification  and  Validation  for  computer  programs  and  associated 
documentation  used  in  embedded  computer  systems  pendiog  revisioo  of  the 
appropriate  Air  Force  regulations.  This  policy  is  designed  to  achieve  the 

i ol lowing  objective: 

Reduce  acquisition  risk  and  increase  software  reliability  and 
maintainability  through  the  use  of  Independent  Verification  and  Validation  on 
embedded  computer  systems. 

3.  Definition: 

a.  Independent  Verification  and  Validation  (IV4V)  (of  computer  programs 
and  associated  documentation)  -  An  independent  assessment  process  structured 
to  ensure  that  computer  programs  fulfill  the  requirements  stated  in  system  and 
subsystem  specifications  and  satisfactorily  perform  the  functions  required  to 
meet  the  user's  and  supporter's  requirements.  IV4V  consists  of  three 
essential  elements:  independence,  verification,  and  validation. 

(1)  Independent  -  an  organizatloo/aency  which  is  separate  from  the 
development  activity  from  a  contractual  and  organizational  standpoint. 

(21  Verification  -  the  evaluation  to  determine  whether  the  products 
of  each  step  of  the  computer  program  development  process  fulfill  all 
requirements  Levied  by  the  previous  step. 

(3)  Validation  -  the  integration,  testing,  and/or  evaluation 
activities  carried  out  at  the  system/subsystem  level  to  evaluate  the  developed 
computer  program  against  the  system  specifications  and  the  user's  and 
supporter's  requirements. 

b.  Emoedded  Computer  Resources  -  Computer  resources  incorporated  as 
integral  parts  of,  dedicated  to,  required  for  direct  support  of,  or  for  the 
upgraaiog  or  modification  of,  major  or  less  than  major  system(s).  (Excludes 
ADP  resources  as  defined  and  administered  under  AFR  300  series.) 
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4.  Applicability  and  Scope:  This  guidance  applies  to  embedded  computer 
resources  in  systems  developed  for  new  acouisitions,  retrofit,  modification, 
or  update  of  existing  systems.  This  policy  applies  to  all  functions  which  are 
iotegral  to  deployed  systems.  Specifically  excluded  are  commercial  computer 
programs  and  associated  documentation  used  io  support  of  program  development 
or  administration,  and  procured  in  accordance  with  Air  Force  300  series 
regulations. 

5.  Development  and  Acquisition  Policy; 

a.  Effective  immediately,  consideration  will  be  given  to  the  use  of  IV&V 
io  new  acquisitions  aod  for  retrofit  or  modifications  of  existing  systems. 

AFSC  aod  AFLC  should  each,  within  90  days  of  the  date  of  this  letter, 
designate  a  single  organization  to  centrally  review  and  coordinate  IV&V  of 
embedded  computer  resource  planning. 

b.  The  designated  computer  program  support  organization  should  be  a  prime 
source  for  accomplishing  an  IV&V.  Due  to  program  constraints  such  as 
schedule,  insufficient  support  resource  availability,  support  organization  not 
yet  designated,  etc.,  it  may  in  some  Instances  be  necessary  to  identify  other 
sources  to  accomplish  the  IV&V.  These  sources  include  qualified  independent 
contractors,  federal  contract  research  centers,  AFSC  laboratories  and  centers, 
product  division  engineering  resources,  Air  Logistics  Center  engineering  and 
technical  resources,  other  military  services  or  Government  agencies,  etc. 
Emphasis  must  be  placed  on  a  qualified  organization/agency  that  has  high  level 
systems  engineering  expertise  and  knowledge  that  goes  beyond  basic  code 
checking. 

c.  An  early  decision  will  be  rendered  regarding  the  extent  of  Independent 
Verification  and  Validation  of  computer  programs  aod  associated  documentation 
acquired  or  supported  under  800-series  regulations.  This  decision  process 
will  be  initiated  as  part  of  the  early  planning  for  computer  resource 
acquisition  or  modification.  This  will  occur  during  concept  formulation  and 
must  be  completed  prior  to  the  release  of  the  full-3cale  development  Request 
for  Proposal  (RFP).  Specifically: 

(1)  The  program/systems  manager  will  convene  the  Computer  Resources 
Working  Group  (CRWG).  The  CRWG  will  use  the  IV&V  criteria  (see  Attachment)  to 
formulate  their  recommendations  as  to  the  extent  and  level  of  IV&V,  and  will 
recommend  a  method  of  accomplishment. 

(2)  The  program/systems  manager  will  use  the  CRWG  recommendations  in 
structuring  a  play  for  risk  abatement  as  reported  io  program  reviews  (e.g., 
PARS,  CARs,  AFSARCs,  DSARCs ,  etc.). 

(3)  Where  IV&V  is  to  be  performed,  the  program/systems  manager  will 
direct  the  CRWG  to  prepare  a  plan  for  accomplishing  IV&V.  This  plan  will 
become  part  of  the  Computer  Resources  Integrated  Support  Plan  (CRISP). 
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(4)  The  program/sy stems  manager  will  ensure  that  tne  RFP  for  the 
system  to  be  developed  includes  appropriate  provisions  to  support  IV&V.  The 
program/sy stems  manager  will  define  IV&V  technical  reaulrements  in  accordance 
with  the  recommended  extent  aod  level  as  defined  in  5c(l)  above.  In  selecting 
tne  IV&V  orgaoization/ageocy ,  the  program/systems  manager  will  consider  tne 
degree  to  which  system/mission  expertise,  rigorous  IV&V  methodology,  aod 
operational  certification  are  reauired. 

6.  Effective  Date:  This  policy  will  be  incorporated  into  AFR  800-H4,  is 
effective  immediately  aod  will  be  reviewed  annually. 

7.  Waivers:  None  authorized. 
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Figure  IV-9-3.  CRITERIA  FOR  EXTENT  OF  IVAV 

The  IVAV  decision  will  be  based  on  the  extent  to  which  computer  programs  could 

affect  the  following  criteria: 

Criteria  Risk 

a.  Safety  H  Failure  of  software  may  cause 

catastrophic  equipment  damage  or 
loss  of  life  includes:  nuclear 
safety,  range  safety,  flight 
safety  of  non-rated  avionics,  air 
traffic  control,  etc. 

M  Failure  of  software  may  contribute 

to  equipment  damage  or  personnel 
hazards  -  includes:  controls  A 
display  indicators  that  may  prompt 
incorrect  commands,  etc. 

L  Failure  of  software  does  not 

affect  personnel  or  equipment 

b.  Mission  Essentiality  H  Potential  error  impact:  mission 

failure 

M  Potential  error  impact:  degraded 

performance 

L  Potential  error  Impact: 

inconvenience 

c.  Technical  Risk  H  Complex,  unproven 

M  Complex;  proven  before  but  not  on 

current  system  or  similar  system 

L  Non-complex;  proven  00  current 

system  or  similar  system 

d.  Supportabllity  H  No  established  support  structure, 

considerable  resources  required 
for  support,  organic  support,  high 
change  frequency  anticipated 

M  Support  concept  broadly  defined 

but  not  specific  to  the  system, 

moderate  support  resources 
required,  organic  support, 
moderate  change  frequency 
anticipated 

L  Support  concept  specifically 

defined,  stable 
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e.  Cost/Scnedule  Impact  H  Large  program;  complex,  on 

critical  path  or  may  become 
critical  path 

M  Small  program;  complex  or 

moderately  complex,  may  or  may  not 
be  on  critical  path 

L  Off-the-shelf  or  non-complex 

f.  Security  H  Potential  unauthorized  access  to 

classified  data  or  unauthorized 
modification  to  CPCI  or  data  base 

M  Inadvertent  loss  or  contamination 

of  classified  data  base 

L  No  classified  data  involved 


g.  Other  criteria  will  be  considered  as  appropriate  by  the  CRWG  or  the 
program/systems  manager. 
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Table  IV-9-U 


OTH-B 


FACTOR 


TOTAL  TV&V 


SCORE  VALUE 
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REFERENCES 

AIR  FORCE  REGULATIONS 


AFR  57-4 
AFSC  Sup  1 

Modification  Program  Approval 

Retrofit  Configuration  Changes 

15  Dec  77 
1  Apr  74 

AFR  65-3 
AFSC  Sup  1 

Configuration  Management 

Configuration  Management 

11  Jul  74 
25  Jul  75 

AFR  66-12 

Aircraft  and  Missile  Equipment  Accountability 

15  Aug  78 

AFR  70-15 
AFSC  Sup  1 

Source  Selection  Policies  and  Procedures 

Source  Selection  Policies  and  Procedures 

16  Apr  80 
18  Feb  77 

AFR  80-14 
AFSC  Sup  1 

Test  and  Evaluation 

Test  and  Evaluation 

12  Sep  80 
19  Feb  81 

AFR  80-45 
AFSC  Sup  1 

Distribution  Statements  on  Technical  Documents 
Distribution  Statements  on  Technical  Documents 

26  Mar  71 
22  May  80 

AFR  300-10 
AFSC  Sup  1 

Computer  Programming  Languages 

Computer  Programming  Languages 

15  Dec  76 
2  Sep  80 

AFR  310-1 
AFSC  Sup  1 

Management  of  Contractor  Data 

Management  of  Contractor  Data 

30  Jun  69 
11  Mar  74 

AFR  800-2 
AFSC  Sup  1 

Acquisition  Program  Management 

Program  Management 

14  Nov  77 
18  Oct  74 

AFR  800-4 
AFSC  Sup  1 

Transfer  of  Program  Management  Responsibilities 
Transfer  of  Program  Management  Responsibilities 

15  Jun  82 
11  May  75 

AFR  800-8 

Integrated  Logistics  Support  Plan 

7  Feb  80 

AFR  800-14, 
AFSC  Sup  1 

Vol  I 

Management  of  Computer  Resources  in  Systems 
Management  of  Computer  Resources  in  Systems 

12  Sep  75 
8  Aug  77 

AFR  800-14, 

Vol  II 

Acquisition  and  Support  Procedures  for 

Computer  Resources  in  Systems 

26  Sep  75 

AFR  800-19 

System  or  Equipment  Turnover 

27  May  75 

AFR  800-21 

Interim  Contractor  Support  for  Systems  and 
Equipment 

26  Sep  78 

AFSC  Sup  1 


Interim  Contractor  Support  for  Systems  and 
Equipment 


28  Jan  80 
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AFSC  Regulations  &  Pamghlets 


AFSCR/AFLCR  80-17 

Air  Force  Engineering  Responsibility  for 
Systems  and  Equipment 

15  Jul  77 

AFSCR  310-1 

Management  of  Contractor  Data 

11  Mar  74 

AFSCR/AFLCR  800-2 

Management  Multi-Service  Systems,  Programs, 
and  Projects 

4  Sep  73 

AFSCP  800-3 

A  Guide  for  Program  Management 

9  Apr  76 

AFSCP  800-7 

Configuration  Management 

1  Dec  77 

Military  Standards  A 

Handbooks 

AFSC  Design 

Handbook  4-2 

Electronic  Systems  Test  &  Evaluation 

10  Apr  71 

MIL-HDBK-334 

Evaluation  of  a  Contractor's  Software 

Quality  Assurance  Plan 

15  Jul  81 

MIL-STD-483 
and  Notice  2 

Configuration  Management  Practices  for 

Systems,  Equipment,  Munitions,  and 

Computer  Programs 

31  Dec  70 

MIL-STD-490 

Specification  Practices 

30  Oct  68 

MIL-STD-1521A 

Technical  Reviews  A  Audits  for  Systems, 
Equipment,  and  Computer  Programs 

1  Jun  76 

MIL-STD-1679 ( NAVY ) 

Weapon  System  Software  Development 

1  Dec  78 

MIL-S-52779A 

Software  Quality  Assurance  Program 

Requirements 

1  Aug  79 

ESD  DOCUMENTS 


ESD-TR-75-85  ADA016488 

ESD-TR- 76-1 59  ADA027051 

ESD-TR-77-16  ADA035924 

ESD-TR-77-22  ADA0371 15 

ESD-TR-77-130  ADA038234 


ESD-TR-77-254  ADA047308 

ESD-TR- 7 7-2 55  ADA047318 
ESD-TR-77-263  ADA048577 
ESD- TR- 77- 32 6  ADA053039 
ESD-TR- 77-32  7  ADA053040 
ESD-TR- 78-117  ADA052567 


An  Air  Force  Guide  for  Monitoring 
Software  Development  Status 

An  Air  Force  Guide  to  Software 
Documentation  Requirements 

Statement  of  Work  Preparation 

Life  Cycle  Events 

Software  Acquisition  Management  - 
Software  Development  and  Maintenance 
Facilities 

An  Air  Force  Guide  to  Computer  Program 
Configuration  Management 

Software  Quality  Assurance 

Verification 

Validation  and  Certification 
Software  Maintenance 
Reviews  and  Audits 


Sep  1975 

Jun  1976 

Jan  1977 
Feb  1977 
Apr  1977 

Aug  1977 

Aug  1977 
Aug  1977 
Aug  1977 
Oct  1977 
Nov  1977 


ESD-TR-78-139  ADA055573 


An  Air  Force  Guide  to  the  Computer 
Program  Development  Specification 


Mar  1978 


